Automatic Group Sparse Coding

Sparse Coding (SC), which models the data vectors as sparse linear combinations over basis vectors (i.e., dictionary), has been widely applied in machine learning, signal processing and neuroscience. Recently, one specific SC technique, Group Sparse Coding (GSC), has been proposed to learn a common dictionary over multiple different groups of data, where the data groups are assumed to be pre-defined. In practice, this may not always be the case. In this paper, we propose Automatic Group Sparse Coding (AutoGSC), which can (1) discover the hidden data groups; (2) learn a common dictionary over different data groups; and (3) learn an individual dictionary for each data group. Finally, we conduct experiments on both synthetic and real world data sets to demonstrate the effectiveness of AutoGSC, and compare it with traditional sparse coding and Nonnegative Matrix Factorization (NMF) methods. Introduction The linear decomposition of a signal (data vector) using a few atoms of a learned dictionary, or the Sparse Coding (SC) technique, has aroused considerable interests recently from various research fields such as audio processing (Févotte, Bertin, and Durrieu 2009), image denoising (Mairal, Elad, and Sapiro 2008), texture synthesis (Peyé 2009) and image classification (Bradley and Bagnell 2008). Different from traditional spectral decomposition methods such as Principal Component Analysis (PCA) and Singular Value Decomposition (SVD), SC (1) is usually additive, which results in a better representation ability; (2) does not require the learned bases to be orthogonal, which allows more flexibility to adapt the representation to the data set. In many real world applications (e.g., the ones we mentioned above), SC achieves state-of-the-art performance. In traditional SC, each data vector is treated as an individual identity and the dictionary is learned over all these data vectors. Recently, (Bengio et al. 2009) pointed out that the SC procedure is just an intermediate step in creating a representation for a data group. For example, the data vectors could be the image descriptors or images, while one data group could be an image or image group. Clearly, the Copyright c © 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. goal of SC is to learn how an image, not an image descriptor, is formed. Therefore (Bengio et al. 2009) proposed a novel technique called Group Sparse Coding (GSC), which can learn sparse representations at the group (image) level as well as a small overall dictionary (image descriptors). One limitation of GSC is that it can only learn a common dictionary over all data groups. However, there should also be an individual dictionary associated with each data group, which makes those data groups different from each other. For example, in electroencephalogram (EEG) signal analysis when the data measured from several subjects under the same conditions (Lal et al. 2004; Lee and Choi 2009), each EEG signal contains some common as well as event (group) related frequency bands and regions. Moreover, in many cases, we only have the data vectors, while their associated group identities are hidden. In this paper, we propose an Automatic Group Sparse Coding (AutoGSC) method, which assumes (1) there are hidden groups contained in the data set; (2) each data vector can be reconstructed using a sparse linear combination of both the common and group-specific dictionaries. We also proposed a Lloyd’s style framework (Lloyd 1982) to learn both the data groups and those dictionaries. Specifically, it is worthwhile to emphasize the strength of AutoGSC. • AutoGSC can learn hidden data groups automatically. In contrast, traditional GSC needs the data group identities to be pregiven. • AutoGSC can learn an individual dictionary for each group, which contains group-specific discriminative information, while traditional GSC cannot. • AutoGSC can also learn a common dictionary for all the groups as traditional GSC. The rest of this paper is organized as follows. Section 2 introduces some notations and related works. The detailed algorithm and analysis is presented in section 3. Section 4 and 5 introduce the experimental results on synthetic and real world data, followed by the conclusions in section 6. Background Without the loss of generality, we assume the data instances are represented as vectors. Mathematically, we denote the observed data matrix as X = [x1,x2, · · · ,xn] ∈ Rd×n, Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence